1. Field of Invention
The present disclosure relates generally to Sensor-assisted Perception of Terrain (SaPOT).
It relates more specifically to computer systems that simulate Sensor-assisted Perception of Terrain. It relates yet more specifically to simulation of real-time SaPOT, where the latter Sensor-assisted Perception of Terrain may be provided, by way of example, for a pilot in an aircraft flying over a terrain of a given topology and a given makeup of materials, where the terrain is illuminated by various sources of radiation (e.g., moonlight, star light) and is thermally-preloaded (e.g., by the daytime sun).
2. Reference to Included Computer Program Listings
This application includes one or more listings of computer programs. The owner of the present application claims certain copyrights in said computer program listings. The owner has no objection, however, to the reproduction by others of such listings if such reproduction is for the sole purpose of studying them to understand the invention. The owner reserves all other copyrights in the program listings including the right to reproduce the computer program in machine-executable form.
3. Description of Related Art
Pilots of aircraft or other pilot-controlled vehicles sometimes guide their craft over a given terrain with the assistance of vision-augmenting equipment.
Examples of vision-augmenting equipment include Night Vision Goggles (NVG""s) and Low Level Light Television Cameras (LLLTV""s).
Such vision-augmenting equipment typically convert hard-to-see imagery in one or more, of the visible and/or invisible spectral bands into imagery that is more clearly visible to the human eye. One example of an invisible spectral band is the Near Infrared Range (NIR) which begins at a wavelength (xcex) of around 0.65 xcexcm and/or slightly longer wavelengths and continues to longer wavelengths. Another example of an invisible spectral band is the Far Infrared Range (FIR) which begins at around 1.00 xcexcm and/or slightly longer wavelengths and continues to longer wavelengths.
LLLTV""s usually amplify photon energy in one or more, of the visible spectral bands (whose xcexc is between around 0.4 xcexcm {violet end} and 0.65 xcexcm {red end}) and sometimes, also downshift and amplify energy in the low NIR band so as to make imagery within those bands more visible to the human eye. The exact bands or sub-bands of operation may vary depending on specific missions. Such missions can be of a law-enforcement nature, or of a military nature, or of another nature. By way of example, a given kind of law-enforcement or agricultural mission might include the seeking out certain types of vegetation whose chlorophyl reflects or absorbs with a specific spectral signature. In such a case, LLLTV sensor response might be controlled to peak in correlation with spectral signature peaks of the targeted vegetation. Searchers will then be able to spot the targeted vegetation more easily.
NVG""s usually operate in the NIR band and produce visible imagery by downshifting wavelength and amplifying. The exact bands or sub-bands of operation may vary depending on specific missions. For example if the mission includes seeking out certain types of objects that are highly reflective of moonlight, the NVG response might be controlled to peak in correlation with spectral signature peaks of the targeted objects.
Forward Looking InfraRed (FLIR) sensors are another example of vision-augmenting equipment. FLIR""s usually operate in a broadband mode that extends across the FIR and NIR bands for producing visible imagery by downshifting wavelength and amplifying. The exact bands or sub-bands of operation and filtering functions may vary depending on specific missions. For example if the mission includes seeking out certain types of very-hot objects that emit black body radiation, the FLIR response might be controlled to peak in correlation with black body emission peaks of the targeted, high temperature objects.
Generally speaking, vision-augmenting equipment use one or more, specially-designed sensors to detect radiations that are either invisible to the human eye or difficult for the human eye to see as distinct imagery. Electronic means or optoelectronic means or other means are provided for converting and/or amplifying the imagery of the sensor-detected radiations into imagery which is clearly visible to the human eye. The sensor-detected radiations may be in the form of reflected energies (e.g., moonlight reflected from IR-reflecting materials) or in the form of directly radiated energies (e.g., IR radiated from a hot automobile engine or from flares), or combinations of both.
Specific uses of vision-augmenting equipment (e.g., specific missions) may occur in combination with specific lighting environments (e.g., moonlight, cloud reflection, etc.) and specific mixes of terrain materials (e.g., IR-reflecting features, UV-absorbing features, etc.). Each unique kind of mission may call for a correspondingly unique mix of sensors, where each sensor may have a respective, application-specific set of frequency responses and/or sensitivities.
Because vision-augmenting equipment often display imagery which is different from that which a pilot may be accustomed to seeing naturally with his or her own eyes, it is often desirable to train pilots ahead of time so that the pilots can correctly interpret what they see with the vision-augmenting equipment when actual flights occur.
One method of training pilots is to let them practice flying over a given terrain, first while looking out-of-window from the cockpit in daylight conditions, and second by repeating the daytime run under night conditions while using the vision-augmenting equipment to view the same terrain.
A more cost-effective method subjects pilots to computer simulations of both Out-of-Window Perception Of Terrain (OwPOT) and Sensor-assisted Perception Of Terrain (SaPOT) under different conditions of lighting, given different terrain topologies, different terrain materials, different sensor types, and so forth.
A conventional OwPOT/SaPOT simulating system will typically have a very large farm of disk drives (or bank of tape drives) that store approximately 1-TeraByte (1012 bytes) of digitized terrain information or more.
This xe2x80x98terrain mapxe2x80x99 information is usually formatted as a collection of three-dimensional polygons (generally triangles) within an x, y, and z Cartesian frame of reference that represents a real world region having dimensions of approximately 1000 kilometers (1000 Km) by 1000 Km (1 Mm) or more. The distance between adjacent vertices of each polygon will typically correspond to a real world distance of about 100 meters (100 m), which inter-vertex distance is, about 10xe2x88x924 times the side dimension of the overall terrain map.
The exemplary 1 Mmxc3x971 Mm area given above for the terrain map may represent a maximal, mission range for a given type of aircraft. The real world area that is represented by a given terrain map can be greater or smaller as may vary with mission specifics. Similarly, the exemplary, 100 m inter-vertex distance given above for each polygon may represent an empirically selected resolution setting. The inter-vertex distance can be greater or smaller as may vary with different and practical map-creating techniques, different mission objectives, and differing capabilities of OwPOT/SaPOT simulating systems to handle maps of higher or lesser resolution.
In a conventional OwPOT/SaPOT simulating system, the following, xe2x80x98per-vertexxe2x80x99 items of digitized information will usually be stored in the terrain map for each vertex of each polygon:(a) the x, y, z spatial coordinates of the vertex point; (b) a three-dimensional normal vector for defining the slope of a material surface at only the vertex point; (c) color values for defining a perceived, reflected or radiant color of the surface material at the vertex point; and (d) texture values for use in adding texture to the interior of the polygon during rasterization.
The disk farm which holds the conventional terrain map is typically connected to an image-rendering computer system. The image-rendering computer system usually includes a paged, main memory, and one or more graphics-rendering pipeline processors coupled to the main memory. Each graphics-rendering pipeline processor generally includes (1) a geometry engine, (2) a rasterizing engine, and (3) a frame buffer.
The geometry engine receives as one of its inputs, 3D polygon information that is paged in from the 3D, vertices-only, terrain map. The geometry engine also receives as inputs, the position, angle and size of a hypothetical viewing window. The geometry engine also receives as inputs, the positions and intensities of one or more light sources that are illuminating the 3D terrain. In response, the geometry engine computationally projects a set of 2D polygons onto the hypothetical viewing window.
These, projected 2D polygons represent a first order, wire-frame estimation of the image that will be viewed through the hypothetical viewing window based on the given lighting conditions and the 3D terrain represented by the inputted terrain map. The viewing window is typically a planar rectangle that is positioned at some angle and position within the 3D reference frame of the terrain map.
Ray tracing is typically employed to create the 2D polygons projection. A backward ray tracing algorithm may trace back along each of, plural and corresponding normal vectors of the viewing window to exposed polygon vertices in the 3D terrain. The backward ray tracing algorithm will typically continue to trace backwardly, along appropriate refection angles, from the exposed 3D polygon vertices, back to one or more, exposed light sources. Then, based on factors such as the traced distance from the viewing plane back to the light source, the reflection angle, and the color attributes of the 3D polygon vertex; the geometry engine will determine how much and what color of light is to be attributed to each 2D vertex point of each polygon on the planar viewing window.
Geometry engines are typically rated for performance speed in terms of how many polygons per second they can process. In other words, a speed value is given indicating how many triangles or other polygons the geometry engine can step through and project onto the viewing window in a given unit of time. Because geometry engines generally perform many complex calculations, their polygons/second processing rate tends to be relatively low.
As explained above, the output(s) of the geometry engine(s) usually represent a first order or wire-frame estimation of the image that will be viewed through the hypothetical viewing window. To improve on this estimation, the 2D output of each geometry engine is transmitted to a rasterizing engine of the same graphics-rendering, pipeline processor. The rasterizing engine adds higher-resolution, texture information into the interior of each two-dimensional polygon that has been projected onto the viewing window by the geometry engine. The texturizing process often includes the steps of: (a) referencing a texture map, (b) pulling appropriate texture data from the texture map in response to information of the surrounding polygon-vertex points and the position within the polygon that is being texturized, and (d) writing that pulled texture data into a corresponding xe2x80x98texelxe2x80x99 (a texture pixel) within the two-dimensional polygon that is being texturized.
Rasterizing engines typically have their performance speeds rated in terms of how many frame-buffer pixels or texels (texture pixels) per second they can process. Generally, the texels/second rate of a rasterizing engine is orders of magnitude larger (e.g., 100 times greater or more) than the polygons/second rate of the corresponding geometry engine. For example, a given geometry engine may process 107 triangles per second while a rasterizing engine of a like technology will process 109 texels per second (2 orders of magnitude greater).
The 2D, texturized imagery that is created by the rasterizing engine is generally stored in a high-speed frame buffer (e.g., video RAM) and eventually passed through to a display system for viewing by a human observer.
The above-description of a typical, image-rendering computer system covers only the generic situation in which it is assumed that only the visible frequency band of light is to be considered and only the response of the human eye to such light is to be considered. This assumption of course, fails to hold true for the Sensor-assisted Perception of Terrain part of a OwPOT/SaPOT simulating system because the SaPOT part of the operation needs to consider multiple frequency bands and the reflection characteristics of different terrain materials in each of these, bands and the cumulative response characteristics of different sensors to all the relevant frequency bands.
A number of tricks are often used in the conventional OwPOT/SaPOT simulating system to make the generic, graphics-rendering, pipeline processor behave as though it were simulating a sensor that is responsive to multiple frequency bands and as though it were simulating various terrain materials that reflect differently in each of these bands. For each frequency band (e.g., NIR, FIR, etc.) or sub-band under consideration, the per-vertex colors of the terrain map are re-encoded so that they mimic the response of a given sensor at the given frequency band and so that they also mimic the reflecting characteristics of the corresponding terrain material which is present at the corresponding vertex point of the respective terrain polygon. Then, for each frequency band, the correspondingly re-encoded terrain map is run through the graphics-rendering, pipeline processor so as to create a band-specific image in the frame buffer for the specific sensor. Multiple passes are generally required, with each such pass handling only one of the bands or sub-bands and also the specific arrangement of light sources that operate in that band or sub-band. A first of the saved, and (sub-)band-specific, frame buffers may define sensor response to the terrain as illuminated by a single point source such as the sun being positioned in a specific part of the sky at a given time of day. A second such frame buffer may indicate a superposable, sensor response due to illumination by a different light source such as the moon (infrared radiation source) being located elsewhere. Yet a third frame buffer may provide a corresponding and superposable image for the sensor when the terrain is illuminated by a diffuse light source such as a clouded sky. A further fourth frame buffer may provide a corresponding superposable image for the sensor when the terrain is illuminated by an artificial light source such as a spectrally-specific flare that is momentarily injected into a specific location in the scene.
The resulting, (sub-)band-specific and sensor-specific, frame buffers are then combined according to the frequency-domain, integrating properties of the specific sensor so as to produce a composite frame. This composite frame is then displayed in a corresponding time slot for viewing by the pilot-in-training.
To produce moving pictures, composite frames need to be composed for respective time slots and then displayed in sequence, one after the next. The pilot-in-training is usually allowed to manipulate a joystick that guides his simulated aircraft through the simulated terrain according to arbitrarily selected, flight maneuvers. Terrain map information is paged into the system""s main memory on an as-needed basis and processed accordingly to try to produce a real-time simulation of what the pilot-in-training would see through the vision-augmenting equipment (e.g., Night Vision Goggles) under the simulated conditions.
The real-time production of composite image frames, where each composite image frame is generated by weighted integration of multiple frame buffers, typically calls for very expensive, and high speed computer hardware (e.g., workstations from Silicon Graphics Inc., where SGI is a well known, computer graphics company located in California). Even when such expensive, and high speed computer hardware is used, the one-(sub-)band at a time, multi-pass operations of the conventional Ow/SaPOT simulating system are so time-consuming that they strain the ability of the system to provide realistic simulations of sensor response. Rendering resolution is particularly limited by the vertex-per-second speed of the geometry engine or engines, which, as explained above, tend to be relatively slow.
Because of the above difficulties, previous sensor-simulating computer systems provided composite images of generally poor resolution and slow speed.
Worse yet, if sensor attributes had to be changed because one wished to simulate the response of a new and different sensor, completely new re-encodings of the entire terrain map generally had to be provided. This meant that extremely large disk farms (or tape drive banks) had to be provided if one wished to be able to quickly switch from simulating a first sensor to simulating a different, second sensor. The problem of requiring large disk farms, with new terrain map encodings for each kind of sensor, compounded to even larger proportions if one wished to store maps of different terrains, one for each of a large number of different sensors.
In view of the above, there has been a long felt but unsolved need in the art of simulating real-time Sensor-assisted Perception of Terrain (SaPOT) to have computer systems and methods that provide better resolution, at higher speed and at less cost. Moreover, for some mission applications (e.g., law enforcement or military), the maintaining of mission secrecy is a concern. Prior methods of maintaining re-encodings of large terrain maps presented serious security problems.
Structures and methods may be provided in accordance with the invention for overcoming the above-mentioned problems. More specifically, means may be provided in accordance with the invention for defining luminance signals for use by a rasterizing engine of a graphics pipeline, where the defined luminance signals account for contributions that may be made to the viewed image due to one or more of the following: (a) sensor response characteristics, (b) terrain materials characteristics, and (c) cumulative lighting sources.
In general, one embodiment of the invention may be seen to comprise a set of specially-configured LUT""s that are used in a rasterizing portion of a graphics system for simulating Sensor-assisted Perception of Terrain (SaPOT). Data flow through the LUT""s is arranged so that simulation of the image produced by a given sensor can proceed rapidly and with good accuracy at a per-texel resolution. More specifically, the set of lookup tables includes a sky-map LUT which is provided for simulating the act of looking up to the sky along the normal surface vector of a given texel to thereby obtain a reading of the sky""s contribution of illumination to that texel. A reflectance-LUT is further provided for simulating the act of reflecting filtered radiation (light) off the material surface of the given texel to thereby obtain a reading of the amount of light that the surface of the texel will reflect. The reflectance and sky contribution factors are multiplied to obtain a per-texel signal representing the amount sensor-detectable light produced from each texel. A generally similar approach is taken to determine what amount of sensor-detectable, black body radiation will be produced from each texel. Reflected radiance and emitted radiance may be added together on a per-texel basis to provide an accurate composite of contributions from both. Each texel is allowed to have a specifiable mixtures of plural surface materials. The mixture factors are also accounted for. Each texel is allowed to be shadowed during different times and the per-texel shadowing is accounted for. As such, a fast but high-resolution system is provided for simulation of SaPOT.
A terrain texels-defining memory may-be provided in accordance with the invention to comprise a plurality of addressable texel records where each such record contains: (a) one or more material identification fields (MID""s) for identifying one or more materials making up the surface of a corresponding terrain texel; (b) one or more mixture fields (MIX""s) for defining mixture proportions for the materials identified as making up the surface of the corresponding texel; and (c) slope-defining data for defining a surface slope or normal of the corresponding texel, where the slope-defining data is preferably, normal vector data that immediately defines a vector that is normal to the surface of the corresponding texel.
A terrain texturizing apparatus in accordance with the invention comprises: (a) a sensor-modulated, current sky-map LUT (lookup table) for receiving per-texel normal vector data and for responsively producing sky radiance data representing a per-texel contribution of illumination from a given sky map, as seen through spectral filtering characteristics of an in-use sensor; (b) a sensor-modulated, reflectance LUT for receiving per-texel surface material identifying data (MID""s) and for responsively producing sensor-modulated reflectance signals (RSM) representing a per-texel reflecting factor due to the surface material make up of the corresponding texel and further due to spectral filtering characteristics of the in-use sensor; (c) a temperature LUT for receiving solar loading data and per-texel surface material identifying data (MID""s) and for responsively producing per-texel temperature signals representing a per-texel temperature due to the surface material make up of the corresponding texel and its thermal history; (d) a black body LUT for receiving the per-texel temperature signals and for responsively producing per-texel emissance signals; (e) a radiance combining element for receiving the per-texel emissance signals of the black body LUT, the sky radiance data of the current sky-map LUT, the sensor-modulated reflectance signals of the reflectance LUT and for responsively producing per-texel and per-material luminance signals.
A machine-implemented, terrain texturizing method in accordance with the invention comprises the steps of, for each to-be-illuminated pixel (texel): (a) identifying_ one or more materials making up the surface of the texel; (b) defining mixture.proportions for materials making up the surface of the texel; (c) defining a vector that is normal to the surface of the texel; (d) using the normal vector to look up into a supplied sky map and to thereby determine an amount of radiance contributed to the texel from the sky map; (e) using spectral filtering characteristics of an in-use sensor to modulate the per-texel amount of radiance contributed to the texel from the sky map; (f) using spectral filtering characteristics of materials making up the surface of the texel to determine black body emissance amounts of the texel; (g) combining the per-texel black body emissance amounts and the per-texel radiance amounts to thereby define a per-texel luminance values attributable to the texel; and (h) using the per-texel luminance values to paint a corresponding pixel in a frame buffer.
A secured terrain simulation method in accordance with the invention comprises the steps of: (a) supplying as separate data signals, (a.1) general terrain vector information; (a.2) sensor-independent sky map .information; (a.3) sensor-independent, terrain overlay texture information; and (a.4) sensor parameters.
A perception of terrain unifying method in accordance with the invention comprises the steps of: (a) providing an Out-of-Window Perception Of Terrain (OwPOT) image on a per-texel basis; (b) providing a Sensor-assisted Perception of Terrain (SaPOT) image on a per-texel basis; and (c) blending the OwPOT and SaPOT images.
A method in accordance with the invention for producing a reflectance LUT for use in conjunction with a specified sensor comprises the steps of: (a) assigning successive and unique material identifications (MID""s) to a set of different materials that can be found in a prespecified, target terrain, the MID""s being translatable into successive and unique addresses of to-be-defined contents of the reflectance LUT; (b) obtaining data defining reflection percentage (Rho) versus wavelength for each of said set of different materials, where the Rho data covers a relevant wavelength range, xcex0 to xcex1 of the specified sensor; (c) integrating the Rho versus wavelength function for each of said set of different materials to thereby define respective, cross-band reflection values (R) for each of the materials; and (d) recording the obtained cross-band reflection values as said addressable contents of the reflectance LUT. This producing method can be refined to provide the recording of sensor-modulated reflection values (RSM) in the reflectance LUT by obtaining data defining relative detection sensitivity versus wavelength for the specified sensor and multiplying such detection sensitivity data with the Rho data on a per wavelength basis during said integration operation.
A method in accordance with the invention for producing a sky-map LUT for use in conjunction with a specified sensor comprises the steps of: (a) defining a coordinate system for pointing to different positions of an illuminating sky that can be found in a prespecified, mission arena, the sky coordinates being translatable into successive and unique addresses of to-be-defined contents of the sky-map LUT; (b) obtaining data defining illumination power density versus wavelength for each of a set of different illuminating sources that can be present in said mission arena, where the illumination power density data covers a relevant wavelength range, xcex0 to xcex1 of the specified sensor; (c) integrating the illumination power density versus wavelength function for each of said set of different sources to thereby define respective, cross-band contribution values (PSD) for a center of each of the sources; and (d) recording the obtained cross-band contribution values (PSD) as said addressable contents of the sky-map LUT. This LUT producing method can be refined to provide the recording of sensor-modulated contribution values (SM-PSD) in the sky-map LUT by obtaining data defining relative detection sensitivity versus wavelength for the specified sensor and multiplying such detection sensitivity data with the illumination power density data on a per wavelength basis during said integration operation.
Other aspects of the invention will become apparent from the below detailed description.